NPS-MAE-04-005 


_  NPS 

PRAtSTANTIA  PERSC/ENTMa^ 


NAVAL 

POSTGRADUATE 

SCHOOL 

MONTEREY,  CALIFORNIA 


DETERMINING  THE  NUMBER  OF  ITERATIONS  FOR 
MONTE  CARLO  SIMULATIONS  OF  WEAPON 
EFFECTIVENESS 

by 

Morris  R.  Driels 
Young  S.  Shin 

April  2004 


Approved  for  public  release;  distribution  is  unlimited. 
Prepared  for:  Defense  Threat  Reduction  Agency 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


REPORT  DOCUMENTATION  PAGE 


Form  Approved  OMB  No.  0704-0188 
Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including 
the  time  for  reviewing  instruction,  searching  existing  data  sources,  gathering  and  maintaining  the  data  needed,  and 
completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any 
other  aspect  of  this  collection  of  information,  including  suggestions  for  reducing  this  burden,  to  Washington 
headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite 
1204,  Arlington,  VA  22202-4302,  and  to  the  Office  of  Management  and  Budget,  Paperwork  Reduction  Project 
(0704-0188)  Washington  DC  20503.  _ _ 

I.  AGENCY  USE  ONLY  (Leave  blank)  2.  REPORT  DATE  3.  REPORT  TYPE  AND  DATES  COVERED 

April  2004  Technical  Report,  March  2003  -  March  2004 

4.  TITLE  AND  SUBTITLE:  5.  LENDING  NUMBERS 

Determining  the  Number  of  Iterations  for  Monte  Carlo  Simulations  of  Weapon 
Effectiveness _ 

6.  AUTHOR(S)  Morris  R.  Driels  and  Young  S.  Shin _ 

7.  PEREORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES)  8.  PEREORMING 

Naval  Postgraduate  School  ORGANIZATION  REPORT 

Dept,  of  Mechanical  &  Astronautical  Engineering  NUMBER  NPS-MAE-04-005 

700  Dryer  Rd,  Monterey,  CA  93943-5000 

9.  SPONSORING  /MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES)  10.  SPONSORING/MONITORING 

Defense  Threat  Reduction  Agency  (DTRA)  AGENCY  REPORT  NUMBER 

Structural  Dynamics  Branch  (TDOS) 

6801  Telegraph  Rd,  Alexandria,  VA  22310 

II.  SUPPLEMENTARY  NOTES  The  views  expressed  in  this  thesis  are  those  of  the  author  and  do  not  reflect  the  official 

policy  or  position  of  the  Department  of  Defense  or  the  U.S.  Government. _ 

1 2a.  DISTRIBUTION  /  AVAll.ABll  JTY  STATEMENT  12b.  DISTRIBUTION  CODE 

Approved  for  public  release;  distribution  is  unlimited. _ 

13.  ABSTRACT  (maximum  200  words) 

Many  weapon  effectiveness  tools  are  implemented  using  a  Monte  Carlo  simulation  approach  since  closed  form  solutions  are 
too  mathematically  intractable  to  compute.  A  question  that  usually  arises  in  connection  with  such  simulations  is  to  ask  how 
many  iterations  of  a  particular  Monte  Carlo  simulation  are  needed.  This  report  proposed  the  probability-based  approach  to 
computing  effectiveness  measures  for  better  feedback  to  the  user  regarding  the  relationship  between  the  number  of  iterations 
executed  and  confidence  measures  associated  with  the  result. 


14.  SUBJECT  TERMS  15.  NUMBER  OF 

Effectiveness  Measures,  Monte  Carlo  Simulation,  Confidence  Limits,  Confidence  Intervals  PAGES  30 


16.  PRICE  CODE 


17.  SECURITY 

18.  SECURITY 

19.  SECURITY 

20.  LIMITATION 

CLASSIFICATION  OF 

CLASSIFICATION  OF  THIS 

CLASSIFICATION  OF 

OF  ABSTRACT 

REPORT 

PAGE 

ABSTRACT 

Unclassified 

Unclassified 

Unclassified 

UL 

NSN  7540-01-280-5500  Standard  Form  298  (Rev.  2-89) 

Prescribed  by  ANSI  Std.  239-18 


1 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


11 


NAVAL  POSTGRADUATE  SCHOOL 
Monterey,  California  93943-5000 


RDML  Patrick  W.  Dunne,  USN  Richard  Lister 

Superintendent  Provost 


This  report  was  prepared  for  Defense  Threat  Reduction  Agency  (DTRA) 

and  funded  by  Defense  Threat  Reduction  Agency  Structural  Dynamics  Branch  (TDOS). 

Reproduction  of  ail  or  part  of  this  report  is  authorized. 


This  report  was  prepared  by: 


Morris  R.  Driels 
Professor  of  Mechanical 
&  Astronautical  Engineering 


Reviewed  by: 


Anthony  J.  Healey 
Chairman 

Department  of  Mechanical 
&  Astronautical  Engineering 


Young  S.  Shin 
Professor  of  Mechanical 
&  Astronautical  Engineering 


Released  by: 


Leonard  A.  Eerrari,  Ph.D. 
Associate  Provost  and 
Dean  of  Research 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


IV 


ABSTRACT 


Many  weapon  effectiveness  tools  are  implemented  using  a  Monte  Carlo  simulation 
approach  since  closed  form  solutions  are  too  mathematically  intractable  to  compute.  A 
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1. 


Introduction 


Many  weapon  effectiveness  tools  are  implemented  using  a  Monte  Carlo 
simulation  approach  since  closed  form  solutions  are  too  mathematically  intractable  to 
compute.  A  question  that  usually  arises  in  connection  with  such  simulations  is  to  ask  how 
many  iterations  of  a  particular  Monte  Carlo  simulation  are  needed?  This  question  may 
only  be  answered  if  it  is  qualified  by  some  performance  measure  expected  of  the 
simulation.  For  example,  a  common  output  of  such  a  simulation  might  be  the  average 
damage  a  target  sustained  when  attacked  by  a  specific  weapon  with  a  known  accuracy  of 
delivery.  If  the  simulation  were  allowed  to  run  for  an  extremely  large  number  of 
iterations  we  would  expect  the  average  damage  to  be  reasonably  accurate  whereas  for  a 
smaller  number  of  iterations  we  would  expect  a  different  result.  The  question  asked 
above  may  be  re-phrased  in  the  form;  how  many  iterations  need  to  be  performed  in  order 
to  obtain  a  specified  accuracy  in  the  result? 

This  issue  is  relevant  to  a  number  of  Weaponeering  tools  currently  used  in  JAWS 
(BAM,  BAS,  TARCOM,  HTM),  JWES  (PVTM,  FBAR)  and  IMEA. 

2.  Population  and  Sample 

The  discussion  above  illustrates  the  concept  of  population  and  sample.  If  a  Monte 
Carlo  simulation  were  run  an  infinite  number  of  times  each  damage  value  computed 
would  be  one  data  point,  x,  of  the  population,  and  we  could  compute  statistics  such  as 
the  mean  px  and  standard  deviation  Ox.  Usually  however  we  cannot  let  a  simulation  run 
an  infinite  number  of  trials  and  we  are  limited  to  just  n  trials,  each  providing  one  damage 
value.  This  limited  collection  of  data  is  known  as  the  sample.  We  can  still  compute  the 
sample  mean,  although  it  is  called  the  average  and  denoted  by  x  ,  and  the  sample  standard 
deviation  Sx.  Clearly  of  interest  is  how  close  are  the  population  and  sample  statistics? 
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3,  Weaponeering  example 


In  order  to  illustrate  coneepts  to  be  discussed  later,  we  will  consider  a  specific 
example.  This  is  the  case  of  a  unitary  target  attacked  by  a  weapon  described  by  a 
Carleton  damage  function  with  known  delivery  accuracy.  It  is  the  example  given  in  the 
writers  Weaponeering  textbook  chapter  7,  and  the  issue  is  to  compute  the  amount  of 
damage  caused  by  a  single  weapon  (SSPD),  where  for  simplicity  only  the  range  direction 
is  considered.  This  example  is  dealt  with  in  detail  in  appendix  A  and  has  the  advantage 
that  a  closed  form  solution  for  the  SSPD  is  mathematically  tractable  as  shown  in  the 
appendix.  Also  shown  in  the  appendix  is  a  Monte  Carlo  simulation  approach  to  solving 
for  the  SSPD,  and  therefore  this  allows  us  to  compare  both  the  closed  form  and 
simulation  approaches  to  computing  weapon  effectiveness.  This  example  should  be 
studied  carefully  before  proceeding. 

The  Monte  Carlo  results  shown  in  the  table  illustrate  one  way  to  determine  how 
many  iterations  are  needed  to  compute  the  correct  SSPD  to  three  decimal  places,  at  least 
for  this  model  and  the  specific  inputs  used.  Two  issues  of  interest  are: 

(i)  Can  we  determine  the  number  of  iterations  needed  without  running  the  model 
many  times? 

(ii)  Given  a  specified  number  if  iterations,  how  close  is  the  computed  damage  to  the 
true  answer? 

Both  of  these  questions  are  addressed  by  estimation  theory  in  general,  and 
confidence  levels  and  limits  in  particular. 

4,  Mean,  Variance  and  Standard  Deviation 

The  mean  of  a  sample  comprising  n  numbers  is  defined  by 

1  «  ^ 

X=-^X,  =-(Xi+X2+...  +  xJ  (1) 

n  n 
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The  variance,  defined  by 


FAR =s/=^x -xr=^[(x,-xr+(x,-xr+...+ (x„  -lyi  (i) 

n-1'- 

The  square  root  of  the  varianee  Sx  is  known  as  the  standard  deviation. 

5,  Confidence  Level,  Confidence  Limits  and  Intervals 

From  tables  of  the  cumulative  distribution  function  for  a  normally  distributed 
random  variable  it  may  be  seen  that  about  68%  of  the  impacts  should  lie  in  the  range  ±a, 
95.5%  will  lie  in  the  range  ±  2g,  and  99.7%  in  the  range  ±  3g.  This  allows  us  to  illustrate 
the  use  of  confidence  limits  and  levels  in  the  following  manner; 

(i)  We  can  be  68%  confident  that  a  random  sample  of  the  variable  x  will  lie  within 
plus  or  minus  one  sigma  of  the  mean. 

(ii)  We  can  be  95.5%  confident  that  a  random  sample  of  the  variable  x  will  lie  within 
plus  or  minus  two  sigma  of  the  mean. 

(iii)  We  can  be  99.7%  confident  that  a  random  sample  of  the  variable  x  will  lie  within 
plus  or  minus  three  sigma  of  the  mean. 

The  percentage  value  is  the  confidence  level,  and  the  interval  within  which  the  value 
of  X  is  expected  to  fall  is  the  confidence  limit. 

This  range  may  be  expressed  in  the  form  of  an  upper  (U)  and  lower  bound  (L) 

where 

U  =  x+z^a  (3) 

and 

L  =  x-  z^(j  (4) 


Values  of  confidence  coefficients  Zc  for  different  confidence  levels  are  given  in  Table  1. 
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Table  1.  Values  of  Zc  for  different  confidence  levels. 


Confidence 

Level  % 

99.75 

99 

98 

96 

95.5 

95 

90 

80 

68 

50 

Zc 

3 

2.58 

2.33 

2.05 

2 

1.96 

1.645 

1.28 

1 

0.6745 

6.  Confidence  Intervals  for  the  Mean 

The  confidence  interval  (for  a  given  confidence  level)  described  above  is  an 
estimation  of  the  variable  x,  a  member  of  the  population.  It  is  also  possible  to  generate 
confidence  intervals  for  the  mean  of  a  population. 

Suppose  we  have  a  normally  distributed  random  variable  x  where  the  population 
mean  is  px  and  variance  Ox  ,  and  a  sample  size  n  is  drawn  from  the  population  where  the 
sample  mean  is  x  and  variance  Sx  .  If  we  do  this  a  large  number  of  times  we  have  a 
distribution  of  x  and  Sx  .  What  can  we  say  about  the  distribution  of  means?  It  is  possible 
to  show  that: 

(i)  The  expected  1  mean  of  x  is  px  the  population  mean 

(ii)  The  expected  variance  of  x  may  be  expressed  in  terms  of  the  variance  of  x, 
and  is  given  by 


n 


We  are  now  able  to  define  a  confidence  limit  and  level  for  the  mean  x  .  If  the 
confidence  level  is  selected  at  say  95%,  then  the  confidence  interval  of  the  mean  is 


(4f?)o.95  1-96  CTj 


(6) 


1  The  usual  interpretation  of  the  expeeted  value  is  the  mean,  therefore  the  expeeted  value  of  the  mean 
reads  “  the  mean  of  the  means” 
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This  is  stated  as:  we  are  95%  confident  that  a  sample  of  the  mean  x  will  be 
within  (L,U)  of  the  true  mean,  or  stated  another  way,  we  are  95%  confident  that  the  true 
mean  is  within  (L,U)  of  a  sample  of  the  mean  x  . 

However,  using  equation  (6)  we  write 

(4^)o.95  1-96  cr^=//j±  1.96^  (7) 

yin 

Since  the  population  mean  and  variance  are  unknown,  the  unbiased  estimators  x 
and  Sx  are  used  giving 

{L,U\,,=x±\.96^  (8) 

yjn 

The  more  general  expression  is 

iL,U)  =  x±z^^  (9) 

yjn 

In  terms  of  the  Weaponeering  test  case  in  the  appendix,  the  variable  x  is  the  value 
of  damage  probability  for  a  single  iteration,  i.e.  selecting  a  random  weapon  impact  point 
and  computing  the  damage  to  the  unitary  target.  The  population  statistics  px  and  variance 
Ox  would  be  obtained  by  repeating  this  process  an  infinite  number  of  times,  where  px  is 
the  SSPD.  If  the  simulation  is  run  for  a  finite  number  if  iterations  (n)  the  sample  statistics 
X  and  variance  Sx  are  estimates  of  the  population  statistics  and  x  is  the  best  estimate  of 
the  SSPD.  We  can  use  equation  (9)  however  to  estimate  the  confidence  interval 
associated  with  a  particular  confidence  level  for  the  SSPD. 

Consider  the  following  example.  Suppose  the  Weaponeering  example  in  the 
appendix  is  run  once  each  for  100,  1000  and  100,000  iterations  and  we  are  interested  in  a 
95%  confidence  level  for  the  mean.  The  results  may  be  summarized  in  the  following 
table. 
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Table  2.  Single  run  of  weaponeering  example. 


n 

X 

s. 

Half  Interval 

100 

0.153 

0.785 

0.154 

1,000 

0.172 

0.302 

0.019 

100,000 

0.177 

0.308 

0.002 

Taking  the  example  for  1000  iterations,  we  may  state  that  we  are  95%  eonfident 
that  the  true  SSPD  (mean)  is  within  0.172±0.019,  i.e.  in  the  range  0.153  to  0.191.  Clearly 
as  we  take  more  iterations  the  half  interval  deereases  and  the  sample  mean  approaehes  the 
population  mean,  and  the  sample  varianee  approaehes  the  population  varianee.  Note  that 
for  a  fixed  number  of  iterations  we  will  in  general  get  different  sample  means  and 
confidenee  limits  for  the  same  eonlidenee  level. 

It  may  be  eoneluded  that  for  any  Monte  Carlo  simulation  of  the  type  deseribed 
above  a  eonlidenee  interval  for  a  speeilied  eonlidenoe  limit  may  be  ealeulated,  but  that 
these  values  apply  only  to  the  simulation  eompleted  and  will  be  different  if  the  simulation 
is  run  again  for  the  same  inputs. 

7.  Confidence  Interval  Estimates  for  Small  Samples  of  the  Mean 

If  the  number  of  samples  of  the  mean  is  small,  typieally  less  than  about  25,  then 
the  eonlidenee  eoeffieients  given  in  Table  1  eannot  be  used.  Instead  of  using  the  normal 
distribution,  the  so-ealled  t  distribution  has  to  be  used.  This  distribution  has  a  similar 
shape  to  the  normal  distribution,  but  is  a  funetion  of  an  additional  parameter  ealled  the 
degrees  of  freedom,  v.  The  probability  density  funetion  for  different  degrees  of  freedom 
is  shown  in  Figure  1. 
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A  f  (X) 


As  may  be  seen,  the  t-distribution  tends  to  the  normal  distribution  as  the  number 
of  degrees  of  freedom  inereases.  In  fact,  the  number  of  degrees  of  freedom  is  determined 
as  the  number  of  samples  minus  one,  so  this  why  for  small  sample  sizes  the  t  distribution 
has  to  be  used  instead  of  the  normal  distribution.  Just  as  the  integral  of  the  normal 
probability  density  function  (the  CDF)  is  tabulated,  so  is  the  integral  of  the  t-distribution. 

The  confidence  level  and  limits  work  the  same  way  as  for  the  normal  distribution. 
Shown  in  Figure  2  is  a  t-distribution  for  a  specific  number  of  degrees  of  freedom  together 
with  a  confidence  limit  (L,U)  for  a  sample  of  the  random  variable  x. 
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For  a  normal  distribution,  the  range  (L,U)  would  be  given  by  equation  (9),  but  for 
a  t-distribution  it  is  given  by 


{L,U)  =  x±t^ 


(10) 


Here  the  mean  and  varianee  are  ealeulated  from  the  small  sample.  As  before  the 
eonfidenee  eoeffieient  h  is  obtained  from  the  area  beneath  the  PDF,  but  this  time  it  will 
dependent  on  the  number  of  degrees  of  freedom  for  whieh  Figure  2  is  drawn.  It  is  usual  to 
tabulate  the  area  under  the  PDF  for  a  single  “tail”  i.e.  tc  is  given  only  for  the  upper 
interval,  U.  What  this  means  is  that  if  we  require  for  example  the  95%  eonfidenee 
interval,  then  the  sum  of  the  area  under  the  PDF  less  than  L  and  grater  than  U  must  be  5% 
of  the  total  area,  henee  the  eonfidenee  eoeffieient  needed  in  equation  (10)  is  to.975.  Table  3 
shows  values  of  the  eonfidenee  eoeffieients  for  different  eonfidenee  levels  and  degrees  of 
freedom. 


Table  3.  Confidence  coefficients  tc  for  t-distribution. 


V 

90% 

95% 

97.5% 

99.5% 

1 

3.08 

6.31 

12.71 

63.66 

2 

1.89 

2.92 

4.30 

9.93 

3 

1.64 

2.35 

3.18 

5.84 

4 

1.53 

21.3 

2.78 

4.60 

5 

1.48 

2.01 

2.57 

4.03 

10 

1.37 

1.81 

2.23 

3.17 

30 

1.31 

1.70 

2.04 

2.75 

00 

1.28 

1.65 

1.96 

2.58 
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From  this  point  on  we  will  assume  we  are  dealing  with  “large”  samples  so  the 
normal  distribution  is  reasonable,  however  this  seetion  provides  the  neeessary  tools  to 
make  the  adjustments  if  the  sample  size  is  small. 

8,  Number  of  Iterations  for  a  Specified  Error  Bound 


Reconsider  the  Weaponeering  example.  The  true  SSPD  is  known  from  the  closed 
form  solution  to  be  0.176,  so  Table  2  could  include  a  column  that  shows  the  percentage 
error  between  the  estimated  SSPD  and  the  true  value. 


Table  4,  Percentage  error  as  a  function  of  iterations. 


n 

X 

%  error 

too 

0.153 

13.1 

1,000 

0.172 

2.3 

100,000 

0.177 

0.1 

It  is  also  noted  that  the  population  standard  deviation  is  not  known,  but  may  be 
accurately  estimated  by  running  the  simulation  a  large  number  of  times,  say  10^.  The 
result  of  doing  this  is 


0.3073 


(11) 


The  question  suggested  by  Error!  Reference  source  not  found.Table  4  is 
whether  we  can  specify  a  maximum  acceptable  percentage  error  for  the  mean  and 
determine  the  required  number  of  iterations. 
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By  considering  the  confidence  interval  to  represent  twiee  this  maximum  error  we 
ean  write 


error„ 


The  pereentage  error  of  the  mean  beeomes 


(12) 


Transforming  for  n  yields 


E  = 


lOOxzA 

x4n 


loo^A 

Ex 


(13) 


(14) 


So,  for  the  example  used  where  the  confidence  level  is  95%,  Zc=0.196,  x  =0.176, 
Sx=0.3073  and  E=5,  the  required  number  of  iterations  becomes  4684. 


In  words  this  reads;  If  the  simulation  is  run  for  4684  iterations,  we  are  95% 
confident  that  the  calculated  SSPD  will  not  differ  by  more  than  5%  from  the  true  SSPD. 


Running  the  simulation  ten  times  for  4684  trials  produced  pereentage  errors  of 
4.89,  4.99,  4.92,  4.92,  5.1,  5.0,  4.95,  4.98,  5.00  and  4.94,  so  it  appears  to  work.  The  only 
problem  is  that  the  separately  calculated  population  statisties  px  and  Ox  were  used  in 
equation  (14)  and  these  are  not  known. 
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9.  Estimating  the  Number  of  Iterations  Required 


Consider  an  iterative  Monte  Carlo  simulation  that  has  no  upper  bound  to  the 
number  of  iterations  to  be  performed.  As  the  iterations  proeeed,  the  damage  estimates 
aceumulate  into  a  sample  of  inereasing  size.  As  more  iterations  take  plaee  the  sample 
approaehes  the  population.  Table  2  shows  the  estimates  of  the  sample  statisties  approaeh 
that  of  the  population.  It  is  proposed  to  ealeulate  the  sample  mean  and  varianee  and  use 
these  values  in  equation  (14)  to  determine  how  many  iterations  are  needed  to  aehieve  a 
speeified  maximum  pereentage  error  with  a  speeified  eonfidenee  level.  It  has  been 
observed  that  this  number  of  iterations  eonverges  quiekly  and  even  for  sample  sizes  an 
order  of  magnitude  lower  than  the  number  required,  the  caleulation  of  that  number  is 
quite  stable. 


Consider  the  previous  example  where  the  exaet  number  of  iterations  was 
determined  to  be  4684.  We  ean  ealeulate  this  number  as  a  function  of  the  number  of 
iterations  actually  performed.  This  estimation  is  shown  in  Figure  3. 
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This  shows  how  the  number  of  iterations  needed  stabilizes  within  about  500-1000 
iterations,  and  within  about  100  iterations  it  is  accurate  to  20%. 

10.  Conclusions  and  Recommendations 

It  is  proposed  that  Weaponeering  programs  that  utilize  Monte  Carlo  simulation 
approaches  to  computing  effectiveness  measures  provide  better  feedback  to  the  user 
regarding  the  relationship  between  the  number  of  iterations  executed  and  confidence 
measures  associated  with  the  result.  Specifically,  selected  programs  can  use  the  methods 
outlined  in  this  paper  to  provide  the  following  user  assistance. 


(1.)  For  a  given  number  of  iterations  give  the  confidence  limits  associated  with  a  user 
supplied  confidence  level(s)  at  the  end  of  the  run. 
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(2.)  While  the  run  is  executing,  provide  the  user  with  the  number  of  iterations  needed 
to  achieve  a  bounded  error  on  the  result  subject  to  a  user  specified  confidence 
level. 


Neither  of  these  features  has  a  computational  overload  of  any  significance  and 
should  be  easily  incorporated  into  any  simulation  for  which  the  source  code  or  interface 
program  is  available.  An  example  of  a  dialog  box  incorporating  the  features  above  is 
shown  in  Figure  4,  and  would  be  displayed  during  program  execution. 


Figure  4,  Suggested  dialog  box  to  display  confidence  levels. 

If  this  dialog  box  were  displayed  during  the  Monte  Carlo  simulation  it  would 
enable  the  user  to  determine  how  accurate  the  simulation  is  so  far,  and  give  an  estimate  of 
how  many  iterations  are  required  to  achieve  a  specific  error  with  known  confidence 
levels.  The  user  may  then  stop  the  simulation  if  sufficient  accuracy  has  been  achieved,  or 
let  it  run  to  a  specific  terminal  error  criterion. 
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